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ABSTRACT. ‘- 


a, 


This paper by. James L. Angel, Director, Medical 
College Admissions Assessment Program, Association of American 
Hedical Colleges, was presented at the 1976 annual meeting of the 
; American Association for the Advancement of Science. Discussed are 
various issues and factors considered in the development of the, nee 
Medical Collége Admissions Test to be introduced’ in 1977, a." ; 
description of the. development project producing the new test, and an 
outline of the CeEEene, and objectives of-the new test., (SL) 
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THE NEW MEDICAL COLLEGE ADMISSIONS TESTS?" 
NEW DIMENSIONS.-IN ASSESSMENT 


“by 2s Pa we 


James L.’ anges f 


The three to'one ratio of applicants to available first year 


ev 2 


in medical school, assures that two of every three candidates will 


or stopped by barriers erected by the admissions process. Admissfons is 
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positions 


be delayed. 


RenphO and cemanas the utmost in energy, taretul thought, and comparison 


- \ 


if ald competitors are’ to be treated panes: 


- * 


One element of the havetax (or gateway for one third) is the 


¢ 


Medical 


College Admission Test- (MCAT). _The significance of: the MCAT is highlighted : 


‘by its suscessful neato as a. cognitive predictor of success in the medical 


: school, and the consequent waltence “admissions committees place on it as a s 


measure of academic waite genane in evlparation for medical, school. 


’ 
yds 


. The power of the MCAT as an admission variable is based on the weight > 
‘ ‘i a / ° . - 


_ given te the scores by committees in relation to grade point averages and .in- 


~dicators of, personal qisdthiad obtained through interview, autobiographies, 


ae “hay 


aca from ailvpanee and other relesaat ‘background’ information. 


y ‘ 
If an 


admissions ioe. relies on cognitive data (MCAT scores, grade point 


averages)" as/ prime ie aia then MCATs and school performance 


‘records as barriers to admission’ become highly Significant factors. If 


personal qualities are studied carefully through available evaluation 


e 


: procedures and balanced in importance with tests and grades, then the impor- 


tance of MCATs as barriers is diminished, and other selection procedures” 


‘. (assessment tec] iques) rise in imp8rtance. 
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I will address theMCAT as an admissions barrier for minorities, but I 


will alter from simply historical p@rspective since the instrument is undergoing . 
$ y ; 


major change, and new tests with new objectives will be introduced in 1977. - 
, °The current MCAT will see its last uge in Fall, 1976. 2 


It is not presumptuous fo say that a’major reason for revising the MCAT 
has to do with concerns about its importance in admissidéns and its ultimate ‘effect 


on minoritias. Social emphasis on the need for quality and availability of 
a V a 


health care are also prime concerns, and admissions officers are constantly 


‘ — 


criticized for "not selecting the right students", thus contributing to current 
-health care problems. I doubt if any of us-would accept this simplistic 


criticism at face value, but it draws attention to the aaa is gateway” 


or barrier played -by the almissiond committee. . : ar: 
Selection is an-imprecise science, but the Association of American, Medical 


Colleges (AAMC) is committed to a’ major effort to improve both cognitive and 
non-cognitive ‘assessment instruments and techniques’ (such as interviews) used 


in admissions. I will discuss the MCAT today, but future reportg out of AAMC 


will reflect the heavy emphasis being placed on assessment of personal ataitige a. 


~ 


at admissions “in relation to expected qualities to be found in the competent, 
concerned physician. 7 ; ys ia a ? : 
The issues discuswed here are based on 1) the MCAT as an admission tool, 
especially as it relhtes to aiaaeiey applicants, and 2) the eitecns now under a 
way at AAMC to improve the dverall gusty of assessment procedures used for 
_ selecting applicants fies positions in medical school, | 
The‘current figh status of the physician's career makes entrance into 


~ * ’ 
\ medical school a prized possession for many people. Statistics are well ‘known \ 


on ber of candidates and number of positions available in médical schools. 


About 45,000 students* file ‘over 300 ,000 applications for about 15, 300 entering 


é ‘ 
, 


tm 


positions each year. ale cabanas nowpetteton Aigtardcetly’ hae moxied to the 


| 
‘ 


disadvantage f iainorities. A disproportionate nusber pf minorities have 


never had. opportunity to ) consider a ohyaictin’ 8 career as a possibility-a 


é 


aeriqus probien not Widutn my Ue to, discuss now ‘but, within the melackatee of 


- this cdaference. aL bypass this significant. séenant of history fo Aighste a ‘tev 


e A} 


‘of nnn deen? ‘taken by AARC to attack the problems created by this inequity and “to 


" open up. new avenues of opportun that might lead to equal opportunity in oY 


education, Yn social mobility, in aspirations, and’ in acqusition.., 


‘ 


. 


_ In the late 60's, when much of our society was being sensitized by the 
social turmoil going on in the country, AAMC bégan to examine thoroughly its 
,role in society, emerging with aggressive, pgsitive responses to the various 


" pressures and opportunities afforded by the climate of the times. An early 


step was creation of ie Office of Minority Affairs, buaded by Dario Prieto, 
which ‘aa hail wide involvement in and impact on ‘babic issues plaguing our 
society in relationship to minority participation. The office has received. 
constant generous support frost the Association’ and its’ constituent Schools, and 
“has taken. firm acytes in peepetian Fe ereceiny social concerns. AAMC's 


. constituents _(the medical schools and keachiag Bospaca}s) have made significant 
1 ’ 
advances in recruiting and ‘eradiing ‘members ‘of minority ae to _become full-. 
~ 


fledged members of the medical profession. ‘Many ‘admigsions ‘committees, have 


attempted to évatuite candidates in a ome will: otic aaa personal sokentlals 


that may be hidden by factors such as slightly "toyer GPA! 8. OF MCAT scores, 


a " : 
cultural-ethni¢ inhibitions, or prejudices of the reviewing committee. 


; e i. } 
The Council of Deans of the Association of American Medical collgbes, 
‘ 7 5 oy + 


through a report of an ad hoc committee convened in 1971, outlined significant 


steps it felt that the Association and its constituents should take in © 


examining ehagreetone process” (AAMC ,, 197]). ‘The pressure of inordinate 


fy 


numbers a REPEC S, concern over ‘minority’ representation in the appli®ant 


cs :. ‘aie 


0 : 
“pool and in ee classes, and a general feeling ,ot concern about the’ 


approach used by schools in evaluating candidates presaced the recommendations. 
. *. AAMC's Division of Educational Measurement and BhaRRrEh, under Dr. Jase 
te aaa ote an immediate study'to determine best ways te respond to this. 
report. A survey with a group of constituents indicated strong support for 
studies to be conducted in ataalond, with MCAT being an important part ok 
this review. From this support, the Medical College Admissions Assessment 


Program (MCAAP) was organized. MCAAP is a broad-based program to stugy. both 


. cognitive and non-cognitive factors ‘of the assessment techniques -used in , ' 


r 


* 


admissions, relating such a study both to the selection’ procedures and -PFe lems 
of prediction encountered in any admiawtons techniques. Through MCAAP, a 

task force was organized from constituents, including deans, admissions officers, ' 
_students, faculty, and researchers in medical schools, as well as ‘preprofessipnal 


advisors and representatives from related agencies sugh as the National Board » . 
’ 2 » a. Se i 
of Medical Examiners and the AMA Liason Committee on Medical Education. This 


task force, under chairmanship of Thomas Meikle, M. D., of Cornell Schéol ‘of 


* . 7 
Medicine, formulated recommendations which became the bases for Executive ~ 


Council action &md guidelines for the MCAAP study. . j . : 


ra 
‘ 
The MCAAP developmental effort has had the continued and extensive 


involyement of minorities and women. - Based ¢ on the premise that the concerns’ 


. and interests of any group will ~ better protested by full involvement of 


that group, MCAAH has been: an open study iacariorartig knowledgeable and 


informed people from its inception. The following review will indicate how * ae 
° ‘ pod . 2 : 


this participation has occurred and.demonstrate the broad reliance 
: ~ . ; 


on members from many groups to work within the study. 
3 oon f f ip é ' 
A full pameetpeton of erojece Meetvities is beyond the scope of this ee 
, ai” 9 , , 
report. The Sntes, ayagen I will cover are: 1) planning for MCAAP projects, : t 


»2) ‘the national noe tatte, and 3): cognitive test deve tapuentss These should 


SaMOREECAES the paints © to be made a regarding barriers ‘to a career as 


a shvwicta, = 8 ‘ : . 


ws e - . 
Initially, guidelines for participation of conettbvants were prepared 


‘and, in eouperation with AAMC's Office of Minority AERA TE; a carefully conceived 


plan fer full involvement of minority members and women was” designed. tthe 


women's plan is not a part of this presentatian. )” 


“i 

1. Through discussions with leaders from Various ‘minority sectors, mostly 

health related, an mn hoc committee for stinsedty’ concerts in MCAAP was formed 
inmediately. Its purpose was to formulate’ a carefully conceived position paper 
that would take into een the ay saad {ssues related Ao minorities' 

access to wedicel echdoi, Nitty apacial: exphaata on admissions screening techniques 
and their impact on minorities., Chaired by Dr. Jobp eteee University e 


California, San Francisco School ' of Medicine, this committee developed: a paper 


that became an insttusiental ‘statement in the deliberations of the MCAAP national my rahi 


task force, _ organized shortly havanttat< - 
The position paper recommended a) seeenuaias of a new st dardized a. + 


test,-b). research to find improved ways to assess pergonal qualities of sppiiteants, 


c): full validation studies of assessment instrum nts, ° ay ‘ueive éénsideration of 

ir - ¢ "7 
a éettaricasretareniad “admissions exam, and e) strengthened counséling and 
advising resources. 


It was proposed that. une science a ‘of . biology, chemistry, and 
pisuten be presented eepuvatels in the new Lasts ena that skills in\handling 
of written materials be assessed. , Peg te oe . 7 ee Se 

For summary , the committee strongly urged a ney test, sates tp the studs 


sith practice of medicine, that ] presents applicants with tasks which fairly’ 


. 


assess their knowledge and skills. sabdenad ‘should be relevant and provide the 


means by yee stodents dencbaitite their Sil for ae school. « 


‘ . tl , 
ia) . 43 
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2.. Through a series of regional conferences and related meetings a. * 
’ P 
national task force emerged. It provided a broad base for constituent 


: put from fadtica? and Graweitent levels. Besides the paper’ described | 
° a) ; 


above, position papers were presented by medical school deans, admissions 


officers, undergraduate advisors, a committee studying personality assessment, j 


s 


and several individuals. er 
& 


Pee The task force was comprised of nearly oye third minority members , 


“@! 


Ps ' Pa . 
assuring not just adequate representation, but a.full cohort of voices to ~ 


" ,debate and discuss issues. *Briefly, the reagmmendations of the task force 
‘ ’ ee 


cover the following: 1) An advisory committee within AAMC will maifttain 
constant gurveilance of admissions processes, to ashuve ongoing research 
” se 3 


and to produce the best possible techniques .and measuring instruments for a, 


5 selection of candidates. This committee has been formed, chafffed. by Cheves 


t - 


Smythe, University of Texas Medical Scahool at flouston. -2) Validation of 


4 ' 4 eed L res? ; : oh 
assessment instruments or techniques is an integral part of MCAAP. This 
: . - a. ‘ a) ° . 
means, that assessment instruments are to.be résearched constantly to 


w datedadice their eta i as selection devices Content, facé; and concurrent ' 


- ovaltaiey aré essential first steps, but a long-term effort will ue launched 
ta study admissions variables ton" their usetuiness as predictors foi success 
in. meet ‘whee. ia practice.: 3) The current MCAT fa decent predictor of © 
success’ in pedical basic sciefices) “a Being used until new tests are teady. 
7) ‘New cognitive padustane ar€é to replace the MGAT. To be introduced in . - * 
t. 


' 1977, they assess a student's ‘ability to work with information in written or br 


. ; quantitative Sora | test “Widwledge and problem aciviag: skills through separate 
ae: Pi hy Uy “ oo et vas J . . ry . ; , 
exams in biology, chemistry, and physics. In. the future, additional areas will 


‘ “pe considered in which students’ may demonstrate proficiency, with behavioral 


eos 


dciencés an early possibility. 5). Assessment of conti dcaliactudl (non-cognitive) 
ee ig. ON ‘y. . 


t ; o. > 


factors of human behavtor Ms the focus of a major proposal at AAMC, which: 
incorporates a research dese’ for the project and suggests a course of action 
for conducting the necessary studies. The extent of this prooeEE will require 
euteide resources to agsute that it can be appropriately carried outs 6) A 
mee information program has been launched, with intent to establish communi- 
cation with admissions officers, applicants, advisots and researchers. An 


. 


initial manual in the cognitive assessment project describes the new exams in . 
detail. Later manuals will present test content and sample questions prior to 

test administration, score interpretation, use of exams in counseling, significant , 
research itcie, and other topics. 

3. The third stage, sdunbetve test development, represents one of the most 
extensive efforts to date to attack the barriers facing students wishing tp 2 


enter medical practice, The MCAAP Task Force Report,clearly stated that every 


‘effort must be made .to provide an assessment process for admissions which is 
, s 


. ‘ ‘ 


. 
"special" treatment has to be accorded 


quitable, does not évaate conditions where 

to any person, and where the assessment of inowhadza and skills is relevant 

to the task at hand. This means program objectives, specific goals, and peourane 

designed to move us”away from concepts such as s disadvantaged, pases 

ssi Broups, reverse dis¢rimination, and other ' phenaes of thequity.. y 
Theovgb fully pledged support of the Executive couned of AAMC, a request 

for proposal was submitted to contractors in this country who could offer 

innbvative ways to seven the problems at issue in cognitive assessment. Five 

major propdsals of striking aualiey vace devatoas in response. Careful ™ 

independent evaluations were conducted, inelwaing avaiusytons from wiscies 

members and women in medicine. The contract went to the American Institutes. 

for Research (AIR) on two greatness: First was their quegestton of an 


ii babble procedure by which important prereqyinitts for the study 


and practice of medicine could be identified . Second was their proposal for 
8. 


: ie : Qa a a 


Ay 


full participation and involvement by underrepresented groups. Their proposal, 


coupled with AAMC's plan for full communication with users, should result in a 


.significant advance in admissions testing. 


The contributions of minorities to all project activities is a key factor 


ee 


“in deyelapasdt of the new exams. Careful construction of PES and 
questions utilizing extensive involvement of minority group members in each phase 
was one significant step. A brief lie ie pg process follows. ~ 


The content upon which aay test is Vaned becomes'the determinant for 
4 ‘ 
content validity. | Validation procedures ‘eunliy involve a set of judgments by 


DS 
informed people who determine the important élementS of knowledge and skills 


which should be assessed as prerequisites to study. The proposal from AIR 


5 


‘expanded on this concept considerably. 
Comprehensive outlines of science knowledge were developed «in biology, 
hi ‘ . 


chemistry and physics. Skills considered important in the study and practice 
f 


of medicine were identified a sted. Types of materials from which most 
information is gleanéd were/ identified and entered in the outline. A survey 
booklet was then organized for all of this materigl, and invitations were extended 


* 4 oie: 
* to over three hundred persons from medical education and practice to serve as 
% C ' ' 


evaluators. Approximately- 160 aftually participated’ in the ratings. Their task 
* . 7 2 : 
. was tg rate knowledge topics, skills, and source materials on the basis of their 
: , . \' 
importance as prerequisites to the*study of medicine, and to rate: the outline 


* 


a“second time, this time evaluating in terms of importance as prerequisites to. ~ 


the practice of medicine. 


, , sf ea ey 
Over ‘one tenth of the raters on this national survey panel were minoritiesv 
e oe . . ie *e 


This meant that the numbers were nearly representative of the gempral poptlation, 


but markedly overrepresented in-relationship to the numbers of minoritids in 


medical education or practice. This latter was deliberate, however, since * 


it was more important to get anh adequate population for reliable ratings, than 
* . . 


it was to have a representative population, 


. 


In brief summary , the evidence shows broad agreement anigng both non-minorities 
and minorities as to. important prerequisites for the study and practice of 
medicine. A full’ fepart on this study will be forthcoming within the next year, 
but the’ importance of this preliminary information rests in the ident ts tegh sot 
of the important prerequisites and in the level of agreement found among the 
clusters of raters analyzed. This gives evidence that there may be a certain 
baie level of science (aridlnile and vartadi types of skills required for the 
study and practice of medicine, regardless of background. 


‘ . 
With this information, AIR has proceeded to develop a set of specifications 4 


‘ 


for the new admissions examination, ‘basing it entirely on knowledge and skilis 
judged by the raters as important in medical ‘€ducation or in practice, Nothing 


elge will end up in the test. Topics were eliminated if they did not obtain & 
; i : . 4 
adequate support from the panelists. Thus there should be nov irrelevant 


material in any of the six tests, using the panel results as a guide. 

Several other steps have been followed in the Rares of the new exams 
to provide ah equitable assessment tool. One has been the wide use of minorities 
as item writers. Another has been extensive review. of every test item by 
’ minorities and women, looking both for cultural and sex. bias. Third, and a 


most significant feature, has been the tryout of items with.several hundred 
students, especially minorities, from undergraduate universities in many 

F - 
parts of the country. Students have had‘opportunity to work the items, evaluate 


them in terms of bias or other standards of quality, and to provide open critiques 


with other students and local university faculty members. On the basis of these 
x 4 { as i 
reviews, AIR has been able to remove items judged by ‘students to be discriminatory, 
‘ 4 
of poor quality, tricky, or.otherwise inappropriate. 


; 10 | ee Ss “3 
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Careful studies ‘are done with all data to uncover indications of any 


problems that might arise with-the exams. Research’ is constant to assure early 
adjustment for any inequities which might appear. Basically, the procedure in 


this project is the most extensive*ever undertaken to develop a standardized 


‘ ‘ : ? 
admissions assessment instrument that will be relevant, technically sound, and 


c 


* equitable, to the extent humanly possible, for every pérson wanting to obtain 


a coveted place in medical school. 


Anxiety for many students reaches inordinate levels as they seek admission 


to medical school. Part of the problem is posed by admission testing, which is 


always a point of apprehension and:puzzle to many. In the case of the MCAT, = 
z : 


Ps e 
students have not been awaré of content other than in the general séhse. Furthermore, 


they have been misdirected by commercial publications which: do not always give 
appropriaté emphasis when describing the tesfs. 
‘Strong support has been given by admissions officers and AAMC staff to 


the concept ‘of providing complete” information to students about content to bee 


- 


. measured. American Institutes for:Research.has based the ‘entire test development 


‘ ‘ | 
piocess on the assumption that students will be told the topics to be covered 


by the test, the types: of skills to be assessed, and the nature bf materials 


from which test: items are to be drawn. .Consequently, a manual is being introduced 
3 + ; 
presenting the complete outline of topics in science: knowledge which students should. 


n 


know, the description and definition of skills to be assessed, and a discussion of 


the types of source materbals to be incorporated. Sample items will be included 


in this manual to acquaint students with types of questions to be on the New MCAT 


‘ 
, 


and to give realistic work experiences that should be of most help in their 
} 
v, i 
preparation. Frankly, it is expected that no student will need to go to a commercial 


organization to obtain preparatory information for the new admissions testing 


F « 
we 


program, AAMC feels'a deep obligation to students, advisors, and admissions 


officers to furnish: the essential ingredients for appropriate use of the new exams. 
: ’ 


ey ee 11 
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Student information will in followed in the winter of 1976-77 by a national 
workshop for admissions officers to introduce’ them to the test, its shoxtig 
procedures, and “its potential for use in admissions. Premedical advisors will 
have access cnadd materials, including instructions for use of the examinations 
in counceling and advising. Technical manuals will be furnished for researchers 
and others interested in the technical quality of rhe tests and how they confoym 
‘to professional standards. . 

Medical schools and individual: researchers from the United States and 
Canada, along with interested parties from other countries, will ie tate to 
take part in validation studies and other research efforts in the new program. 
Evaluation of performance as students ant physicians is an important part of 
the establishment of criteria against which admissions variables are studied. 

\ 

Linking common behaviors along’ the medical education and practice continuun, 
and extending them downward to dannasient of applicants, is an important %task 
yet te be completed. AAMC is presently conducting a major research study which 
show esmevttire to this effort, a longitudinal study originally started with 
medical students of the entering class of 1956. Twenty-eight schools are Layee: 
in this avidy, adaes with a cghort of some 2,500 physicians who graduated en eg 
those schools in 1960. “This cohort is participating in a major. study linking 
the various admissions and nkeeenounoe variables of the 1956-60 period with 
current en ee their practices. th} study promised to bring many 
benefits to the validation research of the current project. > 

The MCAAP Task Force had proposed that a new ‘longitudinal st be started , 
with our current students.~ We believe that the introduction of the new euendttve 
test.as well as the plans for ee assessment of personal qualities of 
candidates should be a Logical starting ppint for a new study. Thus, the. first 


elements, of a comprehensive longitudinal design are being considered which should 


result in a continuing systematic effort to improve the quality of candidate 


selection. ; ; 12 


’ + ; . 
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This presentation does*not make the assumption that the inequities of 
admissions to medical school are all going to disappear with the advent of 
better’ assesement procedures... We take into accoynt that solution-of inequities 


is a major social and personal responsibility. We contend, though, that the 


i of over emphasis ‘on cognitive criteria for admissions has been harmful to 


ny candidates, and especially so for minorities. Improved cognitive instru- 


ments may. assure more equitable measures of candidates, but they cannot remove 


the tendency of certain admission committees to. place too much reliance on 


w 


cognitive information. e . ne Bie 
Most admissions committees work constantly to establish better procedures 

for assessing personal qualities of candiddates, but some ‘observers see a need to + 

take more risks in relationship to committees' judgments about. those qualities. 

.. This maybe the only way in which selection criteria can be expanded to offset 

- the detrimental etext of overreliance on grades and test scores.” 


° 


* Our predictors, whether they be CORAEAENE or non-cognitivé in focus, should - 


be looking. for the nausiel indications of potential--that potentia2 which does 
cli always lend itselt to easy @icision making such as can be done with the 
ranking of Seni ecuians We aubait that the protilils for persons admitted to 
medical school should. some day be, quite parallel to the profile of physicians 
needed to deliver quality health care in this country, 

2 duaptetons are widespread that our physician recruitment and training pyputie 
has not always produced responsive, ethical, compentent physicians. Evidence 4, 
stints thee only in the past secs or so has there béen expanded, diligent 
.  éfforts to remove barriers to medical school and practice for women and 
minorities. The fact that nearly fifty percent of the minority applicants are 
adaitted to medical school tn contragt to 8 one out of three rate of acceptance . 


for all applicants gives concrete evidence of improved attention to the problem. . 


; . Pe 13 a : 
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Further work cect to be done ih recruiting minorites for our applicant 
pool. Studies as recent.as 1972 show. that only 4.7 percent of Black college 
: freshmen were planning professional medical careers (Johnson et al. 1975). 
More recent evidence shows that the applicant pool of minorities seeking entrance 
+o medical sehodl has leveled off, and enrollments show a slight decline. These 
-are problems that go beyond the MCAT. We would hope, however’, that the broad 
involvement Qf women and minorities in the planning and construction of a new 
admission bese. foe peateed school will be one significant step in removing 


““parriers for persons who, first of all, have been ‘denied access, and second, 


are. most needed in order to significantly strengthen the country's health 


care delivery system. 
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